Data Description:
The data contains features extracted from the silhouette of vehicles in different
angles. Four "Corgie" model vehicles were used for the experiment: a double
decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This
particular combination of vehicles was chosen with the expectation that the
bus, van and either one of the cars would be readily distinguishable, but it
would be more difficult to distinguish between the cars.
Domain:
Object recognition
Context:
The purpose is to classify a given silhouette as one of three types of vehicle,
using a set of features extracted from the silhouette. The vehicle may be viewed
from one of many different angles.
Attribute Information:
- All the features are geometric features extracted from the silhouette.
- All are numeric in nature.
Objective:
Apply dimensionality reduction technique – PCA and train a model using
principle components instead of training the model using just the raw data.
Importing Libraries
In [71]:
import warnings
warnings.filterwarnings("ignore")
import pandas as pd
import scipy.stats as stats
import numpy as np
from pandas.api.types import is_numeric_dtype
from statsmodels.stats.proportion import proportions_ztest
import seaborn as sns
from matplotlib import pyplot as plt
from scipy.stats import gamma
from sklearn.preprocessing import StandardScaler
sns.set(color_codes=True)
from sklearn.impute import SimpleImputer
from sklearn import preprocessing
from sklearn.metrics import confusion_matrix, classification_report, precision_recall_fscore_support
from sklearn.decomposition import PCA
from statsmodels.stats.outliers_influence import variance_inflation_factor
import statsmodels.api as sm
%matplotlib inline
Functions for EDA
In [69]:
def Distribution_Continous_Variables(series,color,title):
plt.figure(figsize=(10, 5))
sns.distplot(series, color = color).set_title(title)
def Print_Summary(series,title,var):
print(title)
print('Count = {1}'.format(var,len(series)))
print('Mean of {0} = {1}'.format(var,series.mean()))
print('Median of {0} = {1}'.format(var,series.median()))
print('Mode of {0} = {1}'.format(var,series.mode().values[0]))
print('Skewness of {0} = {1}'.format(var, series.skew()))
print('Excess Kurtosis of {0} = {1}'.format(var,series.kurtosis()))
print(100*"*")
def Coeff_Variation(series,title,var):
print('CV of {0} for {1} = {2}'.format(var,title,(series.std()/series.mean())*100))
def BoxPlot(**kwargs):
plt.figure(figsize=(10, 5))
sns.boxplot(x = kwargs['x'], \
y = kwargs['y'], \
data = kwargs['data'], \
color = kwargs['color'], \
hue = kwargs['hue']).set_title(kwargs['title'])
def ViolinPlot(**kwargs):
plt.figure(figsize=(10, 5))
sns.violinplot(x = kwargs['x'], \
y = kwargs['y'], \
data = kwargs['data'], \
color = kwargs['color'], \
hue = kwargs['hue']).set_title(kwargs['title'])
def CountPlot(**kwargs):
plt.figure(figsize=(10, 5))
sns.countplot(y=kwargs['y'], \
hue=kwargs['hue'], \
data=kwargs['data']).set_title(kwargs['title'])
def calc_vif(X):
vif = pd.DataFrame()
vif["variables"] = X.columns
vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
return(vif)
def backwardElimination(x, Y, significance_level, columns):
numVars = len(x[0])
for i in range(0, numVars):
regressor_OLS = sm.OLS(Y, x).fit()
maxVar = max(regressor_OLS.pvalues).astype(float)
if maxVar > significance_level:
for j in range(0, numVars - i):
if (regressor_OLS.pvalues[j].astype(float) == maxVar):
x = np.delete(x, j, 1)
columns = np.delete(columns, j)
regressor_OLS.summary()
return x, columns
Data Preprocessing
In [4]:
#---importing from csv into datframe---#
df = pd.read_csv("vehicle.csv")
In [5]:
df.head()
Out[5]:
In [6]:
df.describe().T
Out[6]:
In [7]:
df.info(verbose=True)
Observation: All features except 'class' are continous in nature while class is categorical
Find missing values
In [8]:
df.isnull().values.any()
Out[8]:
In [9]:
df.isnull().any()
Out[9]:
Number of null values
In [10]:
df.isnull().sum()
Out[10]:
Imputing using median
In [11]:
#---Reference---#
#--https://towardsdatascience.com/6-different-ways-to-compensate-for-missing-values-data-imputation-with-examples-6022d9ca0779
In [12]:
imp_median = SimpleImputer( strategy='median') #for median imputation replace 'mean' with 'median'
imputed_df = pd.DataFrame(imp_median.fit_transform(df.loc[:, df.dtypes != np.object]))
In [13]:
imputed_df.columns = df.loc[:, df.dtypes != np.object].columns
imputed_df.index = df.loc[:, df.dtypes != np.object].index
In [14]:
imputed_df['class'] = df['class']
In [15]:
imputed_df.isnull().sum()
Out[15]:
Imputation using KNN
This method is sensitive to outliers
In [16]:
#!pip install impyute
In [17]:
import sys
from impyute.imputation.cs import fast_knn
sys.setrecursionlimit(100000) #Increase the recursion limit of the OS
# start the KNN training
imputed_knn =fast_knn(df.loc[:, df.dtypes != np.object].values, k=30)
In [18]:
impute_knn_df = pd.DataFrame(imputed_knn)
impute_knn_df.columns = df.loc[:, df.dtypes != np.object].columns
impute_knn_df.index = df.loc[:, df.dtypes != np.object].index
impute_knn_df['class'] = df['class']
In [19]:
impute_knn_df.head()
Out[19]:
In [20]:
impute_knn_df.isnull().sum()
Out[20]:
In [21]:
print(df.shape)
print(imputed_df.shape)
print(impute_knn_df.shape)
Conclusion: We have around 846 data points and 19 features in total. We have also applied different imputation strategies which we will check one by one
BiVariate Analysis and Conclusion
Below we are using the dataset where empty values have been imputed by median
In [22]:
sns.pairplot(imputed_df, diag_kind='kde')
plt.show()
Below we are using the dataset where empty values have been imputed by values calculated by knn
In [23]:
sns.pairplot(impute_knn_df, diag_kind='kde')
plt.show()
Observations:
- Method of imputation does not have much impact on the distribution and correlation between features as observed by the 2 pairplots above.
- We can observe that features like hollows_ratio, scaled_variance 1 etc. have multiple peaks in their distribution kde graphs that explains the fact that these features might have clusters with different centroid values.
- A few of the features are highly correlated and some of the features may contain outliers.
- We will be using PCA to reduce number of dimensions and to remove multicollinearity from our datasets or to make our features more orthogonal. After PCA we could expect more cloud-like distributions in our scatter plots depicting the fact that our features are more orthogonal.
In [24]:
CountPlot(y = 'class',\
hue = 'class',\
data = imputed_df,\
title = "Count comparison of different classes of vehicles")
In [25]:
print("Percentage of vans = {0:.2f}%".format((imputed_df[imputed_df['class'] == 'van'].shape[0]/imputed_df.shape[0])*100))
In [26]:
print("Percentage of cars = {0:.2f}%".format((imputed_df[imputed_df['class'] == 'car'].shape[0]/imputed_df.shape[0])*100))
In [27]:
print("Percentage of buses = {0:.2f}%".format((imputed_df[imputed_df['class'] == 'bus'].shape[0]/imputed_df.shape[0])*100))
Observation: The data here is imbalanced with 51 % of datasets representing 'car'
Detecting Outliers using IQR method
- We will plot boxplot now to observe if our continous features have outliers or not. The boxplot uses IQR method to detect outliers. Also boxplot would give a visual representation of the five point summary
- We will also plot violinplot along with boxplots to compare the distributions as well.
In [28]:
for col in imputed_df.loc[:,imputed_df.dtypes != 'object']:
BoxPlot(x = 'class',\
y = col,\
data = imputed_df,\
color = 'orange',\
hue = 'class',\
title = 'Boxplot of {}'.format(col))
ViolinPlot(x = 'class',\
y = col,\
data = imputed_df,\
color = 'orange',\
hue = 'class',\
title = 'Violinplot of {}'.format(col))
Observation:
- From above we can observer that there are outliers in datasets covering bus mostly.
- We would try to run a svm model over the dataset without removing outliers and then we would run our model after removing outliers to see if there is a difference
Detecting correlation
In [158]:
plt.figure(figsize=(20, 20))
imputed_df_corr = imputed_df.corr(method='pearson')
ax = sns.heatmap(imputed_df_corr, annot=True, cmap='YlGnBu')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[158]:
Let's remove highly correlated features . We will use the threshold of 0.80 and -0.80 to remove both positively correlated and negatively correlated features
In [159]:
plt.figure(figsize=(12, 10))
imputed_df_corr = imputed_df.corr(method='pearson')
ax = sns.heatmap(imputed_df_corr[(imputed_df_corr >= 0.80) | (imputed_df_corr <= -0.80)], annot=True, cmap='magma')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[159]:
Removing the highly correlated features
In [160]:
upper_triangle = imputed_df_corr.where(np.triu(np.ones(imputed_df_corr.shape),k=1).astype('bool'))
cols_to_drop = [column for column in upper_triangle.columns \
if any((upper_triangle[column] >= 0.80) | (upper_triangle[column] <= -0.80))]
In [161]:
print("We will be dropping the below mentioned columns to remove highly correlated features\n")
print(cols_to_drop)
In [162]:
df_wo = imputed_df.drop(columns = cols_to_drop).copy()
Plotting the correlation matrix again
In [163]:
plt.figure(figsize=(20, 20))
imputed_df_corr = df_wo.corr(method='pearson')
ax = sns.heatmap(imputed_df_corr, annot=True, cmap='YlGnBu')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[163]:
Shape of new dataset
In [164]:
df_wo.shape
Out[164]:
In [165]:
sns.pairplot(df_wo, diag_kind='kde')
plt.show()
Observation:After removing correlated features we can observe that in the above pariplot we can fin very few features that are correlated. Mostly the scatterplots are forming cloud like structures with r value almost 0.
Standardizing the continous features
In [166]:
X = df_wo.copy()
cont_feat = X.loc[:,X.dtypes != 'object'].columns.tolist()
features = X[cont_feat]
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
In [167]:
X[cont_feat] = features
In [168]:
X.head()
Out[168]:
Building SVM model
Train-Test split(70:30)
In [169]:
y = X['class'] #---->Target Variable
X = X.drop(columns=['class']) #----->Dependent Variables
Encoding target variable
In [170]:
le = preprocessing.LabelEncoder()
le.fit(y)
Out[170]:
In [171]:
le.classes_
Out[171]:
In [172]:
y_trans = le.transform(y)
In [173]:
np.unique(y_trans)
Out[173]:
In [174]:
y_trans
Out[174]:
In [175]:
le.inverse_transform(y_trans)
Out[175]:
In [176]:
from sklearn.model_selection import train_test_split
X_train,X_test,y_train,y_test = train_test_split(X,y_trans,test_size=0.3,random_state=42, stratify=y_trans)
Training the model with GridsearchCV using kfold cross validations
In [177]:
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
param_grid = {'C': [0.1, 1, 10, 100, 1000],
'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
'kernel': ['rbf']}
svm = GridSearchCV(SVC(), param_grid, refit = True, verbose = 3, cv=10)
svm.fit(X_train, y_train)
Out[177]:
In [178]:
print(svm.best_params_)
In [179]:
print(svm.best_estimator_)
In [180]:
print("Train Score:")
svm.score(X_train,y_train)
Out[180]:
In [181]:
svm = SVC(kernel='rbf', C=100, gamma=0.01)
svm.fit(X_train,y_train)
print("Test Score:")
print(svm.score(X_test, y_test))
Confusion Matrix
In [182]:
plt.figure(figsize=(10, 5))
y_pred = svm.predict(X_test)
cnf_matrix = confusion_matrix(y_test, y_pred)
p = sns.heatmap(cnf_matrix, annot=True, cmap="YlGnBu" ,fmt='g')
bottom, top = ax.get_ylim()
p.set_ylim(2, 0.0)
plt.title('Confusion matrix for SVM model')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
Out[182]:
In [183]:
print(classification_report(y_test,y_pred,target_names = np.unique(le.inverse_transform(y_trans))))
Since the classification is multiclass and classes are imbalanced we would also calculate micro average or the overall accuracy
In [184]:
precision_recall_fscore_support(y_test,y_pred,average='micro')
Out[184]:
Observation
- The accuracy or micro average score is 93% in the case above and it seems to be a correct metric to measure in case of imbalanced dataset
- We can say that our model seems to be performing well with SVM algorithm after we did basic data preprocessing like removing correlated features at thresholds of 0.80 and -0.80
Using p value to find important features
- Our null hypothesis is that the selected features or dependent variables do not have any effect on our target or independent variable
- We will build a naive regression model and will calculate the p values
- If p value turns out to be higher than our significance level(let's assume 0.05) we will discard the combination of features i.e we will accept our null hypothesis for the given set of features
In [185]:
le = preprocessing.LabelEncoder()
le.fit(imputed_df.iloc[:,-1:])
Out[185]:
In [186]:
x_ols = imputed_df.iloc[:,:-1].values
y_ols = le.transform(imputed_df.iloc[:,-1:])
significance_level = 0.05
selected_columns = imputed_df.columns[:-1]
data, selected_columns = backwardElimination(x_ols, y_ols, significance_level, selected_columns)
In [187]:
data_pval = pd.DataFrame(data = data, columns = selected_columns)
In [188]:
data_pval.shape
Out[188]:
In [189]:
y_ols.shape
Out[189]:
In [190]:
plt.figure(figsize=(20, 20))
df_corr_p = data_pval.corr(method='pearson')
ax = sns.heatmap(df_corr_p, annot=True, cmap='YlGnBu')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[190]:
Standardizing the continous features
In [191]:
X_pval = data_pval.copy()
cont_feat = X_pval.loc[:,X_pval.dtypes != 'object'].columns.tolist()
features = X_pval[cont_feat]
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
In [192]:
X_pval[cont_feat] = features
In [193]:
X_pval.head()
Out[193]:
Building SVM model
In [194]:
from sklearn.model_selection import train_test_split
Xp_train,Xp_test,yp_train,yp_test = train_test_split(X_pval,y_ols,test_size=0.3,random_state=42, stratify=y_ols)
Training the model with GridsearchCV using kfold cross validations
In [195]:
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
param_grid = {'C': [0.1, 1, 10, 100, 1000],
'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
'kernel': ['rbf']}
svm = GridSearchCV(SVC(), param_grid, refit = True, verbose = 3, cv=10)
svm.fit(Xp_train, yp_train)
Out[195]:
In [196]:
print(svm.best_params_)
In [197]:
print(svm.best_estimator_)
In [198]:
print("Train Score:")
svm.score(Xp_train,yp_train)
Out[198]:
In [199]:
svm = SVC(kernel='rbf', C=100, gamma=0.01)
svm.fit(Xp_train,yp_train)
print("Test Score:")
print(svm.score(Xp_test, yp_test))
Confusion Matrix
In [200]:
plt.figure(figsize=(10, 5))
y_pred = svm.predict(Xp_test)
cnf_matrix = confusion_matrix(yp_test, y_pred)
p = sns.heatmap(cnf_matrix, annot=True, cmap="YlGnBu" ,fmt='g')
bottom, top = ax.get_ylim()
p.set_ylim(2, 0.0)
plt.title('Confusion matrix for SVM model')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
Out[200]:
In [201]:
print(classification_report(yp_test,y_pred,target_names = np.unique(le.inverse_transform(y_ols))))
Since the classification is multiclass and classes are imbalanced we would also calculate micro average or the overall accuracy
In [202]:
precision_recall_fscore_support(yp_test,y_pred,average='micro')
Out[202]:
Observation
- The accuracy or micro average score is 97% in the case above and it seems to be a correct metric to measure in case of imbalanced dataset
- We can say that our model seems to be performing well with SVM algorithm after we reduced dimensions based on p value instead of simply removing correlated features above and below the thresholds of 0.80 and -0.80 respectively
Now we will do PCA to reduce multicollinearity and try to reduce the number of dimensions in our dataset
Plotting correlation matrix again for the original imputed data
In [203]:
plt.figure(figsize=(20, 20))
imputed_df_corr = imputed_df.corr(method='pearson')
ax = sns.heatmap(imputed_df_corr, annot=True, cmap='cubehelix')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[203]:
Standardizing the continous features
In [204]:
X1 = imputed_df.copy()
cont_feat = X1.loc[:,X1.dtypes != 'object'].columns.tolist()
features = X1[cont_feat]
scaler = StandardScaler().fit(features.values)
features = scaler.transform(features.values)
In [205]:
X1[cont_feat] = features
In [206]:
X1.shape
Out[206]:
In [207]:
X1.head()
Out[207]:
In [208]:
y1 = X1['class'] #---->Target Variable
X1 = X1.drop(columns=['class']) #----->Dependent Variables
le = preprocessing.LabelEncoder()
le.fit(y1)
y1_trans = le.transform(y1)
np.unique(y1_trans)
Out[208]:
Covariance Matrix
For covariance matrix we should standardize our columns
In [209]:
cov_mat = np.cov(X1.T)
cov_mat
Out[209]:
Plotting covariance
In [210]:
plt.figure(figsize=(20, 20))
ax = sns.heatmap(cov_mat, vmax=1, square=True,annot=True,cmap='cubehelix')
plt.title('Correlation between different features')
bottom, top = ax.get_ylim()
ax.set_ylim(bottom + 0.5, top - 0.5)
Out[210]:
Using PCA from scikit learn
In [211]:
pca = PCA().fit(X1)
plt.plot(np.cumsum(pca.explained_variance_ratio_))
plt.xlim(0,7,1)
plt.xlabel('Number of components')
plt.ylabel('Cumulative explained variance')
Out[211]:
In [212]:
np.cumsum(pca.explained_variance_ratio_)
Out[212]:
Observation: From above graph we can observe around 5 principal components capture about 95% of the variance in the data. Hence we can drop components 6 and 7
In [213]:
sklearn_pca = PCA(n_components=5)
X_sklearn = sklearn_pca.fit_transform(X1)
In [214]:
X_sklearn
Out[214]:
In [215]:
X_sklearn.shape
Out[215]:
Plotting pairplot between principal components to visualize correlation
In [216]:
sns.pairplot(pd.DataFrame(X_sklearn), diag_kind='kde')
plt.show()
Observation: From above we can clearly see that our features do not show much correlation and are mostly normally distributed
In [217]:
#----keeping the random state same so that the same rows get picked up for test cases as in case of our last model training with normal features-----#
from sklearn.model_selection import train_test_split
X1_train,X1_test,y1_train,y1_test = train_test_split(X_sklearn,y1_trans,test_size=0.3,random_state=42, stratify=y1_trans)
In [218]:
from sklearn.model_selection import GridSearchCV
from sklearn.svm import SVC
param_grid = {'C': [0.1, 1, 10, 100, 1000],
'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
'kernel': ['rbf']}
svm_pca = GridSearchCV(SVC(), param_grid, refit = True, verbose = 3, cv=10)
svm_pca.fit(X1_train, y1_train)
Out[218]:
In [219]:
print(svm_pca.best_params_)
In [220]:
print(svm_pca.best_estimator_)
In [221]:
print("Train Score:")
svm_pca.score(X1_train,y1_train)
Out[221]:
In [222]:
svm = SVC(kernel='rbf', C=100, gamma=0.01)
svm.fit(X_train,y_train)
print("Test Score:")
print(svm_pca.score(X1_test, y1_test))
Confusion Matrix
In [223]:
plt.figure(figsize=(10, 5))
y1_pred = svm_pca.predict(X1_test)
cnf_matrix = confusion_matrix(y1_test, y1_pred)
p = sns.heatmap(cnf_matrix, annot=True, cmap="YlGnBu" ,fmt='g')
bottom, top = ax.get_ylim()
p.set_ylim(2, 0.0)
plt.title('Confusion matrix for SVM model using Principal components as features')
plt.ylabel('Actual label')
plt.xlabel('Predicted label')
Out[223]:
In [224]:
print(classification_report(y1_test,y1_pred,target_names = np.unique(le.inverse_transform(y1_trans))))
Since the classification is multiclass and classes are imbalanced we would also calculate micro average or the overall accuracy
In [225]:
precision_recall_fscore_support(y1_test,y1_pred,average='micro')
Out[225]:
Final Conclusion
In [229]:
from prettytable import PrettyTable
x = PrettyTable()
x.field_names = ["Algorithm", "Dimensionality Reduction technique","Best train micro average F1 score","Best test micro average F1 score"]
x.add_row(["SVM","Pearson correlation","0.97","0.93"])
x.add_row(["SVM","Hypothesis testing(p value)","0.99","0.97"])
x.add_row(["SVM","PCA","0.88","0.84"])
print(x)
- The test accuracy or test micro average score after PCA is 84% and it seems to be a correct metric to measure in case of imbalanced dataset
- We can say that our model does not seem to be performing as well as it performed when we used original features and used p value method for dimensionality reduction.
- Hence we can conclude that introducing principal components as the feature does not seem to improve the model performance.